Using Disk Throughput Data in Predictions of End-to-End Grid Data Transfers

Data Grids provide an environment for communities of researchers to share, replicate and manage access to copies of large datasets. In such environments, fetching data from one of the several replica locations requires accurate predictions of end-to-end transfer times. Predicting transfer time is significantly complicated due to the involvement of several shared components such as networks, disks, etc., in the end-to-end data path each of which experiences load variations that can significantly affect the throughput. Of these, disk accesses are rapidly growing in cost, and have not been previously considered, although on some machines they can be up to 30% of the transfer time. In this paper, we present techniques to combine observations of end-to-end application behavior and disk I/O throughput load data. We develop a set of regression models to derive predictions that characterize the effect of disk load variations on file transfer times. We also include network component variations and apply these techniques to the logs of transfer data using the GridFTP server, part of the Globus ToolkitTM. We observe up to 9% improvement in prediction accuracy when compared with approaches based on past system behavior in isolation.